gh-148105: _pyrepl: switch console refresh to structured rendered screens by pablogsal · Pull Request #146584 · python/cpython

pablogsal · 2026-03-28T20:25:24Z

Issue: _pyrepl: switch console refresh to structured rendered screens #148105

pablogsal · 2026-03-31T18:40:40Z

Crap I activated this copilot nonsense by mistake ...

Copilot

Pull request overview

This PR refactors _pyrepl screen refresh to operate on structured “rendered screens” (cells/lines + cursor) rather than raw string lists, with corresponding updates to console backends and tests.

Changes:

Introduces structured rendering/layout primitives (RenderCell/RenderLine/RenderedScreen, LayoutMap, content fragments) and updates Reader to produce/compose them.
Rewrites Unix/Windows console refresh() paths to diff/apply RenderedScreen updates via explicit refresh plans.
Updates tests and ancillary behavior (e.g., control-char display escaping and history-file NUL sanitization) to match the new rendering model.

Reviewed changes

Copilot reviewed 21 out of 21 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
Lib/_pyrepl/render.py	New rendering model (cells/lines/screens), diffing, and line-update primitives.
Lib/_pyrepl/layout.py	New layout mapping between buffer positions and rendered rows/columns.
Lib/_pyrepl/content.py	New prompt/body content modeling and buffer-to-fragment conversion.
Lib/_pyrepl/reader.py	Major refactor: invalidation model, rendered screen composition, and layout-driven cursor mapping.
Lib/_pyrepl/unix_console.py	Refresh rewritten to plan/apply structured line updates; fixes raw accumulation bug.
Lib/_pyrepl/windows_console.py	Refresh rewritten to plan/apply structured line updates with rendered state sync.
Lib/_pyrepl/console.py	Console API now refreshes with `RenderedScreen`; adds rendered-state sync + redraw visualization.
Lib/_pyrepl/utils.py	Adds display-char iterator/control escaping helpers; refactors `disp_str`; adds `StyleRef`.
Lib/_pyrepl/types.py	Expands shared type aliases (cursor/dimensions/keymap/event data).
Lib/_pyrepl/trace.py	Adds `trace_text()` helper for safer trace output.
Lib/_pyrepl/readline.py	Sanitizes embedded NULs when reading history; updates overlay invalidation hooks.
Lib/_pyrepl/commands.py	Converts “dirty” flag usage to targeted invalidations; updates screen sync behavior.
Lib/_pyrepl/completing_reader.py	Moves completions menu to overlay-based composition; updates invalidations.
Lib/_pyrepl/historical_reader.py	Updates history operations to invalidate buffer/prompt appropriately.
Lib/_pyrepl/simple_interact.py	Switches interrupt path to full invalidation.
Lib/_pyrepl/input.py	Cleans up TYPE_CHECKING guard for type-only imports.
Lib/test/test_pyrepl/support.py	Test utilities updated for rendered-line equality and new console refresh signature.
Lib/test/test_pyrepl/test_reader.py	Adds/adjusts tests for control-byte escaping, zero-width behavior, and layout mapping.
Lib/test/test_pyrepl/test_unix_console.py	Adds regression test for colorized multiline typing redraw behavior.
Lib/test/test_pyrepl/test_windows_console.py	Updates expected cursor movement/write patterns due to new refresh diffing.
Lib/test/test_pyrepl/test_pyrepl.py	Adds history/NUL sanitization test and multiline history navigation behavior updates.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Lib/_pyrepl/reader.py

Lib/_pyrepl/content.py

johnslavik

I like the new easter egg rainbow mode 🌈.

I'll be spelunking for a while, but here's what I found on the spot.

Lib/_pyrepl/reader.py

johnslavik · 2026-04-05T12:24:12Z

Lib/test/test_pyrepl/test_render.py

@@ -1,104 +0,0 @@
-from unittest import TestCase


Was removing the test_render module intended? It still passes.

Yeah mot sure what happened here. Fixed

Hm, I don't see it yet

Lib/_pyrepl/render.py

Introduce render-cell, rendered-screen, and line-diff helpers for redraw work. Keep the new abstractions self-contained so later commits can adopt them cleanly.

Teach the reader and terminal backends to refresh from RenderedScreen objects. This isolates the redraw-planning refactor before layout and styling changes land.

Add structured prompt, content, and wrapped-row helpers for screen calculation. Move reader layout bookkeeping onto those helpers before styling changes arrive.

Thread StyleRef and styled content fragments through render and reader paths. Keep semantic color information attached to cells during redraw diffs.

Make buffer, layout, prompt, overlay, and full redraw causes explicit in Reader. The invalidation matrix is still cursed, but at least now it is visible.

Move completion and message UI onto overlays layered over the base rendered screen. The completion menu stops pretending it is the base reality.

Finish the refactor with prompt-cell caching, typed aliases, and layout edge-case fixes. Land the remaining render and reader regressions needed for the final shape.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Initialize the refresh cache with safe defaults and recompute lxy from the buffer when the cached screen is reused for overlay-only refreshes. This also adds an assertion to catch buffer changes that bypass cache invalidation. Fix incremental rebuild boundaries by avoiding the empty-source fast path when rebuilding from offset 0, and clear the correct trailing lines in the Unix and Windows refresh planners when diffing from a non-zero offset. While here, remove an unused layout field, keep combining characters attached to their base character in render diffs, and fold in a few small _pyrepl cleanups and comments.

johnslavik

I think I'll need one more round, but I promise I'm getting there. This is awesome.

johnslavik · 2026-04-06T22:10:21Z

Lib/_pyrepl/commands.py


        with self.reader.suspend():
            self.reader.msg = _sitebuiltins._Helper()()  # type: ignore[assignment]



Exiting help mode (from F1) regressed (video right hand side), this seems to be a fix:

Suggested change

self.reader.invalidate_prompt()

Screen.Recording.2026-04-06.at.22.59.54.mov

johnslavik · 2026-04-06T22:39:59Z

Lib/_pyrepl/reader.py

+    def with_cursor(self) -> RefreshInvalidation:
+        if self.needs_screen_refresh:
+            return self
+        return replace(self, cursor_only=True)
+
+    def with_buffer(self, from_pos: int) -> RefreshInvalidation:
+        current = from_pos
+        if self.buffer_from_pos is not None:
+            current = min(current, self.buffer_from_pos)
+        return replace(self, cursor_only=False, buffer_from_pos=current)
+
+    def with_prompt(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, prompt=True)
+
+    def with_layout(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, layout=True)
+
+    def with_theme(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, theme=True)
+
+    def with_message(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, message=True)
+
+    def with_overlay(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, overlay=True)
+
+    def with_full(self) -> RefreshInvalidation:
+        return replace(self, cursor_only=False, full=True)


Suggested change

def with_cursor(self) -> RefreshInvalidation:

if self.needs_screen_refresh:

return self

return replace(self, cursor_only=True)

def with_buffer(self, from_pos: int) -> RefreshInvalidation:

current = from_pos

if self.buffer_from_pos is not None:

current = min(current, self.buffer_from_pos)

return replace(self, cursor_only=False, buffer_from_pos=current)

def with_prompt(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, prompt=True)

def with_layout(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, layout=True)

def with_theme(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, theme=True)

def with_message(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, message=True)

def with_overlay(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, overlay=True)

def with_full(self) -> RefreshInvalidation:

return replace(self, cursor_only=False, full=True)

def with_cursor(self) -> Self:

if self.needs_screen_refresh:

return self

return replace(self, cursor_only=True)

def with_buffer(self, from_pos: int) -> Self:

current = from_pos

if self.buffer_from_pos is not None:

current = min(current, self.buffer_from_pos)

return replace(self, cursor_only=False, buffer_from_pos=current)

def with_prompt(self) -> Self:

return replace(self, cursor_only=False, prompt=True)

def with_layout(self) -> Self:

return replace(self, cursor_only=False, layout=True)

def with_theme(self) -> Self:

return replace(self, cursor_only=False, theme=True)

def with_message(self) -> Self:

return replace(self, cursor_only=False, message=True)

def with_overlay(self) -> Self:

return replace(self, cursor_only=False, overlay=True)

def with_full(self) -> Self:

return replace(self, cursor_only=False, full=True)

johnslavik · 2026-04-07T00:31:46Z

Lib/_pyrepl/console.py

+    def begin_redraw_visualization(self) -> str | None:
+        if "PYREPL_VISUALIZE_REDRAWS" not in os.environ:
+            return None
+
+        palette = self._redraw_debug_palette
+        cycle = self._redraw_visual_cycle
+        style = palette[cycle % len(palette)]
+        self._redraw_visual_cycle = cycle + 1
+        trace(
+            "console.begin_redraw_visualization cycle={cycle} style={style!r}",
+            cycle=cycle,
+            style=style,
+        )
+        return style


Surprisingly often syntax highlighting colors blend into redraw visualization styles:

If it is worth it to fix it (I guess it is?), the low-effort path would be to require true color and pick a more unique debugging palette.

If we must keep 8 bits though (naaah do we?), we could have a separate object tracking just next color index, skipping one color on collision detected while applying the plan. This would avoid accidentally merging adjacent but separate redraw visualizations. A practical assumption would be that _colorize.Syntax palette is always 8-bit, so it would be a matter of checking whether bg_sgr - 40 == fg_sgr - 30.

Other ideas include just resetting foreground color.

johnslavik · 2026-04-07T00:40:58Z

Lib/_pyrepl/layout.py

@@ -0,0 +1,226 @@
+from __future__ import annotations


Do we want to cover this with a dedicated test_layout test suite?

pablogsal changed the title ~~_pyrepl: fix multiline history recall and sanitize NULs in history files~~ _pyrepl: switch console refresh to structured rendered screens Mar 28, 2026

pablogsal force-pushed the pyrepl-new branch 3 times, most recently from 7c26e18 to 3ed6987 Compare March 30, 2026 23:19

pablogsal marked this pull request as ready for review March 31, 2026 18:39

pablogsal requested review from ambv and lysnikolaou as code owners March 31, 2026 18:39

bedevere-app bot added the awaiting core review label Mar 31, 2026

pablogsal assigned pablogsal and ambv Mar 31, 2026

pablogsal requested review from Copilot and removed request for ambv and lysnikolaou March 31, 2026 18:40

Copilot started reviewing on behalf of pablogsal March 31, 2026 18:41 View session

Copilot AI reviewed Mar 31, 2026

View reviewed changes

Lib/_pyrepl/reader.py Show resolved Hide resolved

Lib/_pyrepl/content.py Outdated Show resolved Hide resolved

pablogsal added the skip news label Apr 4, 2026

pablogsal changed the title ~~_pyrepl: switch console refresh to structured rendered screens~~ gh-148105: _pyrepl: switch console refresh to structured rendered screens Apr 4, 2026

bedevere-app bot mentioned this pull request Apr 4, 2026

_pyrepl: switch console refresh to structured rendered screens #148105

Open

johnslavik self-requested a review April 4, 2026 18:17

pablogsal force-pushed the pyrepl-new branch 2 times, most recently from 8843c85 to 95b0c62 Compare April 4, 2026 19:39

johnslavik reviewed Apr 5, 2026

View reviewed changes

Lib/_pyrepl/reader.py Outdated Show resolved Hide resolved

Lib/_pyrepl/reader.py Show resolved Hide resolved

johnslavik reviewed Apr 5, 2026

View reviewed changes

pablogsal force-pushed the pyrepl-new branch from 95b0c62 to a57dc2b Compare April 5, 2026 15:42

pablogsal added 5 commits April 5, 2026 20:19

Add _pyrepl render primitives

41e8b78

Introduce render-cell, rendered-screen, and line-diff helpers for redraw work. Keep the new abstractions self-contained so later commits can adopt them cleanly.

Switch _pyrepl consoles to rendered screens

8ed00a8

Teach the reader and terminal backends to refresh from RenderedScreen objects. This isolates the redraw-planning refactor before layout and styling changes land.

Extract _pyrepl content and layout helpers

65c8481

Add structured prompt, content, and wrapped-row helpers for screen calculation. Move reader layout bookkeeping onto those helpers before styling changes arrive.

Preserve style metadata in _pyrepl cells

bc71f4d

Thread StyleRef and styled content fragments through render and reader paths. Keep semantic color information attached to cells during redraw diffs.

Refactor _pyrepl refresh invalidation state

654385c

Make buffer, layout, prompt, overlay, and full redraw causes explicit in Reader. The invalidation matrix is still cursed, but at least now it is visible.

pablogsal and others added 5 commits April 5, 2026 20:21

Separate _pyrepl overlays from base rendering

fa45e48

Move completion and message UI onto overlays layered over the base rendered screen. The completion menu stops pretending it is the base reality.

Tighten _pyrepl prompt caching and follow-ups

ae551ea

Finish the refactor with prompt-cell caching, typed aliases, and layout edge-case fixes. Land the remaining render and reader regressions needed for the final shape.

Update content.py

3883037

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

fixup! Fix _pyrepl incremental refresh and render edge cases

a1026c0

pablogsal force-pushed the pyrepl-new branch from a57dc2b to a1026c0 Compare April 5, 2026 19:25

Merge branch 'main' into pyrepl-new

1e79b00

johnslavik reviewed Apr 7, 2026

View reviewed changes


		with self.reader.suspend():
		self.reader.msg = _sitebuiltins._Helper()() # type: ignore[assignment]

Uh oh!

Conversation

pablogsal commented Mar 28, 2026 • edited by bedevere-app bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pablogsal commented Mar 31, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

johnslavik left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

johnslavik Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

pablogsal Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

johnslavik left a comment

Choose a reason for hiding this comment

Uh oh!

johnslavik Apr 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnslavik Apr 6, 2026

Choose a reason for hiding this comment

Uh oh!

johnslavik Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

johnslavik Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pablogsal commented Mar 28, 2026 •

edited by bedevere-app bot

Loading

johnslavik left a comment •

edited

Loading

johnslavik Apr 6, 2026 •

edited

Loading

johnslavik Apr 7, 2026 •

edited

Loading